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PREFACE 


The work described in this report was performed by the Advanced 
Technical Studies Office of the Jet Propulsion Laboratory. This report was 
presented as a paper at the Human Factors Society’s 18th Annual Meeting in 
Huntsville, Alabama, on October 1 5- 1 7, 1974, 
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ABSTRACT 


The role of vision in teleoperation has been recognized as an important 
element in the man-machine control loop. In most applications of remote 
manipulation, direct vision cannot be used. To overcome this handicap, the 
human operator's control capabilities are augmented by a "tele" -vision sys- 
tem. This medium provides a practical and useful link between workspace 
and the control station from which the operator performs his tasks. 

The function of the video system is to reproduce the original scenes in 
pictorial form. Systematic errors in terms of photometry, resolution, 
geometry and perhaps color can be removed by decalibration procedures. 
Human performance deteriorates when the images are degraded as a result 
of instrumental and transmission limitations. Recovering images from 
various degradation effects is commonly referred to as restoration. Image 
enhancement is used to bring out selected qualities in a picture to increase 
the perception of the observer. 

At the Image Processing Laboratory (IPLr) of JPL, we employ a general 
purpose digital computer (IBM 360/44) utilizing an extensive special purpose 
software system (VICAR) to perform an almost unlimited repertoire of 
processing operations. This approach has proven to be most flexible, ver- 
satile and suitable for experimental work. 

Guided by the experience of the IPL and the recent advances in L SI 
technology, we are reporting on special hardwired algorithms which have 
speeded up the processing by several orders of magnitude. Although quantum 
limited imaging was made possible by noise removal and contrast enhancement 
as part of a development in electron microscopy, these methods and experien- 
ces are transferrable to other teleoperator applications. The processing and 
enhancement of images are controlled by the operator/scientist matching his 
perceptual needs to optimally adjust the instrument. CentraL to the near real 
time image processing is a high speed digital solid state mass memory 
operating at input/output speeds compatible with standard TV rates. Thus, 
the operator, as the most important link in the loop, is provided with a real 
time interactive display which enables him to perceive the remote workspace 
as required to execute remote manipulation tasks. 
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MAN- MACHINE INTERACTIVE IMAGING AND DATA- PROCESSING USING 
HIGH-SPEED DIGITAL MASS STORAGE 


INTRODUCTION 

With the processing complexity of space 
mission objectives and spacecraft system require- 
ments, the need for interactive and adaptive 
remote manipulator systems has become unavoid- 
able (Ref. 1). Potential control applications are 
anticipated for extending orbital missions, explora- 
tion of planets, docking and repair of orbiting 
spacecraft, or long-term deep space missions. 
These controls will include the ability to perform 
work in near real time. Fast control reaction 
will be required for example: in docking, cargo 

handling, in- space assembly and deployment, 
operation and maintenance of individual space- 
craft and vehicle subsystems. The problems and 
costs to perform such functions nearly in situ 
are formidable. A more effective solution 
to these problems is to situate man in a well 
established and comfortable working environment 
and provide the facilities to remotely perceive and 
affect the space environment. In order to make 
this possible, machines and interfaces must be 
made available to amplify man’ s sensory, cogni- 
tive and effector capabilities. Depending on the 
task requirements, these capabilities can only be 
provided by various levels of autonomous action 
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characteristics. It is safe to speculate that sys- 
tems of variable autonomy will emerge which 
provide a flexible division of man and machine 
responsibilities. This approach effectively will 
protect man from the hostile environment while 
optimizing the use of his unique abilities for per- 
ception, cognition, goal setting, strategy planning, 
and performance supervision in getting "work 
done" , 

In any such system, one of the most impor- 
tant links between man, machine and the environ- 
ment is through the visual sensory subsystem. 

The ideal visual subsystem provides, for most 
tasks, a vital element in establishing a sense of 
presence for the human operator. It conveys 
information for observing the environment and for 
planning of actions as well as feedback information 
for monitoring and evaluating the results of pre- 
vious actions. At the present level of technology, 
visual subsystems are limited in terms of achiev- 
ing these goals (Ref. 2), 

THE "TELE ''-VISION SYSTEM 

Present technology provides augmentation 
of human vision by means of television systems. 


1 



This medium can be used as a practical and useful 
link between work space and the control station 
(Fig. 1) from which the operator performs his 
task* 

Assuming that we wish to transmit the 
image from workspace and display it to the opera- 
tor as a television picture, ideally we expect to 
see a picture which is a replica of the point pro- 
jection of the 3D scene onto a two dimensional 
image plane. This replication means that the 
scene contrast and brightness levels are main- 
tained on pictorial display. There should be no 
geometric degradation in terms of line or picture 
element structure or noise and loss of fine detail. 

Available television imaging systems realis- 
tically do not reproduce the scene faithfully. Un- 
fortunately in searching the literature (Ref. 3) one 
finds that subjective evaluation of television pic- 
tures has been rather limited in extent and that no 
concerted effort has been made first to examine 
seriously the human factors and then to design a 
pictorial data processing system taking them into 
account. Most subjective testing was done after 
systems have been built. M. Baldwin (Ref. 4) 
seems to be one of the first to do any subjective 
testing, where he investigated the effect of image 
sharpness as function of focus adjustment, noting 
subjective reaction to change in image sharpness. 

Co It man and Anderson (Ref. 5) have inves- 
tigated noise limitation? to resolving power in 
electronic imaging displays by using electronically 
generated bar patterns displayed together with a 
noisy background, F. A. RosellandR. H. Wilson 
(Ref. 4) continued this work by considering the 
effects of a finite aperture of real TV type sensors. 

IMAGE DISTORTION IN TV SYSTEMS 


The point image projection transmitted by a 
television system is likely to produce images at 
the display which may be distorted to amplitude, 
shape or phase, or all three. These distortions 
are introduced by the objective lens, optical and 
electrical defocussing, electron scanning beams at 
the sensor and display, bandwidth limitations, 
etc. The overall effect appears as a reduction of 
image sharpness or a progressive attenuation of 
the higher spatial frequencies. This degradation 
is analogous to a one dimensional low pass 


electrical filter which is often referred to as the 
system modulation transfer function or the spatial 
frequency transformer. In imaging two dimensions 
are involved. They are either independent or 
possess radial symmetry; this means that the sets 
of information can be treated separately or become 
one dimensional in character respectively. The 
effective apertures of the individual system com- 
ponents have been found to be additive whereby the 
response to one input stimulus is identical to the 
sum of the responses produced individually. This 
is characteristic for linear systems. It is there- 
fore possible to decompose complex input signals 
by the use of Fourier analysis to sine wave func- 
tions. In the mathematical description, the sys- 
tem modulation transfer function F(S) in the spatial 
frequency domain is expressed by the Fourier 
transform (Ref. 9) from real space g(k, y) to 
frequency space F(w x , uy). It is defined as 

F(S) = F(w x , w y ) = 5f[g(x, y)] 

= & x [g(x)] *& y [g(y)] 


f(x, y) exp [- j{« x x + w y y)] d x 

■00 


where 3F denotes the taking of the Fourier trans- 
form and g(x, y) is an imaging having brightness 
g at the spatial position (x, y) and F(w x , co y ) is a 
vector of amplitude F with frequency components 

co and co . 
x y 

IMAGE RESTORATION 

Considering the typical spatial frequency 
response of a TV System (Fig, 2), spatial filter- 
ing can be used to correct or equalize the overall 
system response. These methods have been used 
extensively for digital image processing (Ref. 7 
and 8). Ideally restoration and degradation are 
inverse functions, which, from the mathematical 
point of view, may be expressed as 

F(i) = F(S) * F(0) 


environment 


WORK SPACE 



Figure 1. M TELE 11 VISION system 
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NOISE LIMITATIONS 



Figure 2. Typical system frequency 
response 

and 


F<0) 


F(i) 

F(S) 


where F(i) is the Fourier transform of input 
image function, F(S) is the transform of the sys- 
tem modulation transfer function, and F(Q) is the 
output or displaced picture transform. The con- 
ditions for correct image restoration are met 
when F(i) = F(0) or the transform of the output is 
identical to that of the input and 


F <°> 


Image restoration may be accomplished by operat- 
ing in the frequency domain, multiplying of the 
Fourier transforms of the picture ( F (i ) ) with the 
corrective filter (Fig. 3), the inverse of the mod- 
ulation transfer function (l/F(S)) and then by 
reinversion of the resulting compensated spectrum 
back to real (image) space. This method has been 
extensively used by digital processing system 
(Fig. 4), The practical limitation to system 
aperture equalization is essentially set by the 
random noise contained within the system band 
pass. 



MHz 


Figure 3. Corrective filter spatial frequency 
characteristic for equalization of 
system shown in Figure 2 


The dependence of image detectivity and 
recognition on the random noise level has been 
treated extensively by Coltman {Ref. 5), Johnson 
(Ref. 10), Rose (Ref, 1 1 ) and Rosell (Ref. 6). If 
image restoration is to be used, the effects on the 
signal to noise ratio in the displayed picture must 
be considered. 

White noise usually originates from the 
statistical fluctuations in the signal, and back- 
ground photoelectrons, and the noise generated by 
the video preamplifier of the TV camera. The 
power spectrum of white noise is of uniform ampli- 
tude and frequency distribution. Therefore, in a 
system which has a given aperture response, the 
signal-to-noise ratio decreases from the maximum 
value at very low spatial frequencies to one at the 
limiting resolution (Fig. 5). Utilizing a low pass 
filter for image restoration (Fig. 6) does not 
improve the signal to noise ratio although the 
modulation transfer characteristic has been 
equalized. Therefore, equalization of the system 
modulation transfer characteristic enhances 
images only when the signal-to-noise ratio of the 
imaging system is high. JFL has recognized this 
problem and decided to improve the signal-to- 
noise ratio of images first and then proceed with 
further processing, 

DIGITAL IMAGE INTEGRATOR 

CIT/Hale Observatories have made avail- 
able, on loan, a digital image recorder which has 
been built around a high speed digital memory 
system. JFL has used the system to demonstrate 
high speed noise reduction as means of image 
enhancement. 

The digital image recorder (DIR) has been 
described in detail by Sachs, H. G. (Ref. 12) and 
Dennison, E. W. (Ref. 13), The DIR accepts 
standard video signals from a television camera. 

A high speed analog to digital converter is used to 
quantize the image brightness levels at a 4. 8 MHz 
sampling rate. In this configuration the image is 
interrogated as an array of 256 x 256 picture ele- 
ments (pixels) and each elemental photoelectron 
charge is transferred into a 65, 536 word memory. 
Each word consists of 16 binary bits, or has a 
dynamic range of 65, 536. The memory consists 
of dynamic MOS shift registers, which are 
accessed sequentially in synchronism with the 
read out of the sensor. The DIR control system 
was arranged to operate as "multiple frame 
grabber". 

The "frame grabber" concept centers around 
the ability of the DIR to operate as a digital image 
buffer. The memory array size dictates how many 
picture elements can be stored and the pixel clock 
indicates the data rate at which the information is 
accessed. Since the memory addressing scheme 
is arranged in the manner of raster scanned TV 
systems, it is easy to utilize television terminology 
which identifies pictures as frames. Due to the 
access speed and digitization rates the name 
"frame grabber" was only natural. By control of 
the array address timing, it is possible to input 
one frame of data into memory. The dynamic 
shift registers, of which the memory is comprised, 
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Figure 4a. Surveyor spacecraft footpad on lunar surface before restoration 



Figure 4b. After image restoration (note improvement in resolution shown by wedges) 
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SIGNAL TO NOISE RATIO IN dB (20 LOG 





Figure 5. Signal to noise ratio as a function 
of spatial frequency 


provide a nondestructive readout of the frame 
stored. This process operates at a frame rate of 
60 per second which guarantees a flicker free 
display on standard TV monitors. 

By continuous application of the video signal 
to the input of the DIR, the system will operate as 
a multiple additive frame grabber or integrator. 
The frame integration time, or the number of 
frames accumulated, is controlled by an acquisi- 
tion timer* How quickly the memory is filled 
(saturates) depends on the signal levels received 
from the camera system. 

To illustrate: 


Maximum bit 
count 

Highlight in image 
Maximum integra- 
tion time 
Maximum number 
of frames per 
integration 


65, 536 counts 

64 counts (64 bits ) 
17 seconds 

1024 


The interesting and significant aspect, of frame 
integration is the improvement in signal-to-noise 
which can be materialized. In the above example, 
the signal to noise ratio could be improved by a 
factor of (number of frames integ rated) 1/2 which 
would be 32. The picture illustrated by Fig. 7a 
shows a single quantum noise limited electron 
micrograph frame. The same image integrated 
for 2 seconds, or 120 frames, is seen in Fig. 7b 
and shows a signal to noise improvement from 
1:1 to approximately 10.5:1 while the theoretical 
improvement factor should have been 11* The 
Fourier transforms corresponding to Fig* 7 are 
shown pictorialiy by Fig. 8 and graphically by 
Fig. 9- 


Figure 6. Noise current as a function 
, of frequency (S/N-40DB) 

(a) Non- equalized system 

(b) Fully equalized system 


CONCLUSION 


Television systems can be used to augment 
man f s capabilities in the control loop of remote 
manipulation systems. This can be accomplished 
despite instrumental and transmission system 
limitation normally causing signal deterioration. 
Experimentation to evaluate various image enhance- 
ment techniques can he performed by computer 
based Image Processing Laboratories such as 
JPL’s IPL System IBM 360/44. This approach 
has been considered most flexible, because it 
permit b the use of an almost unlimited repertoire 
of operations. Unfortunately, in systems where 
operations depend on the responses of man in a 
control loop, even with the fastest machines con- 
ceivable, the general digital computer approach 
is too slow. For repetitive enhancement and 
restoration, hardware approaches offer very 
positive solutions. We have demonstrated that 
high speed frame integration can be used to im- 
prove the signal to noise ratio in near real time 
imaging. The hardware design was made possible 
by the .advances in LSI Technology, which have 
sped up processing by orders of magnitude. Signal 
to noise improvements within a few seconds 
improve the otherwise noise limited resolution and 
provide an excellent base for image restoration 
and contrast enhancement. Most significantly, 
the operator as the most important link in the con- 
trol loop, can be provided with a near real time 
display, which he controls interactively, thus 
enabling him to perceive the remote work space 
as required for the timely execution of remote 
manipulation taskB, 
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Figure 7. Images at output of DIR 

(a) Single frame - S/N - 1 

(b) Integrated frames - S/N - 11.5 
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Figure 8. Fourier transforms of images 

(a) Single frame, corresponding to Figure 7. a 

(b) Integrated frames, corresponding to Figure 7.b 
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Figure 9, Graphical representation of Fourier 
transforms corresponding to 
Figures 7 and 8 * 


ACKNOWLEDGEMENTS 

We would like to acknowledge the support 
provided by Dr. E. Dennison of CIT by the loan 
of the Digital Image Recorder. 

This work was supported by the National 
Aeronautics and Space Administration Contract 
NAS7-100 and the National Institutes of Health 
Research Resource Grant No. RR00443. 


REFERENCES 

1. Deutsch, S. and Heer, E. , "Manipulator 
Systems Extend Man* a Capabilities in 
Space" Astronautics and Aeronautics, 

June 1972. 

2. Alsberg, H. "Visual Perception and Illusion 
in Active Image Transmission Systems" 
Display Update 1 73, December 8, 1972. 


3. Wilder, W. C., "Properties of the Human 
Visual System as an Image Evaluator" 

Short Course, Purdue University, June 
1972., West Lafayette, Ind. 

4. Baldwin, Jr., M. W. , "The Subjective 
Sharpness of Simulated Television Images", 
Bell System Tech. Journal, Vol. 19, 

pp. 563-587, October 1940. 

5. Coltman, J. W. , Anderson, A. E., "Noise 
Limitations to Resolving power in Elec- 
tronic Imaging" Proceedings IRE Vol. 48, 
p. 858-855, May I960. 

6. Rosell, F. A., Wilson, R. H. , "Per- 
formance Synthesis (Electro -Optical 
Sensors)" Westinghouse Defense and Space 
Center, Technical Report AFAL-TR-71- 
137, May 1971. 

7. Nathan, R. , "Spatial Frequency Filtering" 
in B. Sacks -Lipkin et al, eds. Picture 
Processing and Psychopictorics Academic 
Press, New York 1970, p. 151-163, 

8. Nathan, R. , "Picture Enhancement for 

the Moon, Mars, and Man", in G. C. Chang 
et al eds. Pictorial Pattern Recognition, 
Washington 19^8, p. 239-266. 

9. Goodman, J. , "Introduction to Fourie r 
Optics", McGraw Hill, New York 1968. 

10. Johnson, J. , "Analysis of Image Forming 
Systems, 11 Proceedings of the Image 
Intensifies Symposium , pp. 249-273, held 
at Fort Belvoir, Virginia, October 1958, 
sponsored by the U. S. Army Engineer 
Research and Development Laboratories, 
Fort Belvoir, Virginia. 

11. Rose, A,, JOSA, 38, February 1948, 

12. Sachs, H. G. "A Digital Image Recorder" 
Publications of the Astronomical Society 
of the Pacific, Vol. 84, No. 497, 

February 1972. 

13. Dennison, E. W., "Digital Hardware and 
Computer System for a Digital Image 
Recorder", Proceedings of a Symposium 
held by the Institute of Astronomy and 
Space Science, May 17, 1973. 


8 


JPL Technical Memorandum 33-717 




